Global phase diagram of a doped Kitaev-Heisenberg model 
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The global phase diagram of a doped Kitaev-Heisenberg model is studied using an SU{2) slave- 
boson mean-field method. Near the Kitaev limit, p-wave superconducting states which break the 
time-reversal symmetry are stabilized as reported by You et al. [Phys. Rev. B 86, 085145 (2012)] 
irrespective of the sign of the Kitaev interaction. By further doping, a d-wave superconducting state 
appears when the Kitaev interaction is antiferromagnetic, while another p-wave superconducting 
state appears when the Kitaev interaction is ferromagnetic. This p-wave superconducting state does 
not break the time-reversal symmetry as reported by Hyart et al. [Phys. Rev. B 85, 140510 (2012)], 
and such a superconducting state also appears when the antiferromagnetic Kitaev interaction and the 
ferromagnetic Heisenberg interaction compete. This work, thus, demonstrates the clear difference 
between the antiferromagnetic Kitaev model and the ferromagnetic Kitaev model when carriers are 
doped while these models are equivalent in the undoped limit, and how novel superconducting states 
emerge when the Kitaev interaction and the Heisenberg interaction compete. 

PACS numbers: 71.27.-|-a, 74.20.-z, 75.10.Kt 



I. INTRODUCTION 

There has been considerable attention paid to the Ki- 
taev model whose ground state is a gapless Z2 spin liquid 
(SL).^ If such a model is realized, fault tolerant quantum 
computations can be possible. 

The Kitaev model consists of local (iso) spins S* = 1/2 
on a honeycomb lattice as 
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Here, the spin component 7 depends on the bond specie 
as shown in Fig. [T] A2lr03 (A=Li or Na) have been pro- 
posed as possible candidates to realize the Kitaev model 
as Ir^+ ions having the effective angular momentum 
je// = 1/2 form the honeycomb lattice.^ In fact, if the 
correlation effects are strong enough to realize a Mott in- 
sulating state, the low-energy electronic state is described 
by the combination of the anisotropic Kitaev interac- 
tion [Eq. ([T])] and the symmetric Heisenberg interaction, 
Hj = JHj2{rr')^r' Sp' , Called the Kitaev-Heisenberg 
(KH) model. Alternatively, density-functional-theory 
calculations for Na2lr03 predicted the quantum spin Hall 
effect;^ Later experimental measurements for Na2lr03 
confirmed a magnetic long-range order with a "zigzag" 
antiferromagnetic (AFM) patterni^i^ As this magnetic 
pattern is not realized in the model first proposed for 
Na2lr03, where the Kitaev interaction was introduced 
as a ferromagnetic (FM) interaction {Jk < 0) and the 
Heisenberg interaction was introduced as an AFM inter- 
action {Jh > O),*' the importance of additional contri- 
butions such as longer-range magnetic couplings2r2£ and 
lattice distortions^^ were suggested. Recently, the sign 
of Kitaev and Heisenberg terms was reconsidered^^ by 
including the direct hybridization between neighboring 
Ir t2g and Cg orbitals,^'^ It is found that, when the Ki- 
taev interaction is AFM and the Heisenberg interaction 
is FM, zigzag-type AFM ordering could be stabilized in 



FIG. 1: (Color online) Schematic view of the Kitaev- 
Heisenberg model. 7 = a;, J/, 2 in the left figure show the spin 
components for the Kitaev interaction. rx,y,z are unit vectors 



connecting the nearest-neighbor sites. 
Brillouin zone is shown. 



On the right, the first 



accordance with the experimental report. 

While the Kitaev SL state is not realized in Na2lr03, 
there could appear novel states by carrier doping if this 
system is described by the KH model. Specifically, con- 
sidering the FM Kitaev- AFM Heisenberg model, such an 
effect was studied in Refs. and Both studies found 
the triplet (p) superconductivity (SC) by carrier dop- 
ing, but the SU (2) slave-boson mean-field (SBMF) study 
found a state which breaks the time-reversal symmetry 
(termed p SCi),J^ while the U{1) SBMF study found a 
time-reversal symmetric state (termed p SC2).^^ Exotic 
triplet pairing was also suggested from a low-energy ef- 
fective model for layered cobaltateJS Recently, artificial 
bilayers of perovskite transition-metal oxides (TMOs) 
grown along the [111] crystallographic axis were pro- 
posed as new platforms to explore a variety of quantum 
effects, It was pointed out^^ that such a bilayer involv- 
ing Srlr03 (Ref. X9) could also realize the KH model 
when the correlation effects are strong enough to yield a 
Mott insulating state. But, both the Kitaev interaction 
and the Heisenberg interaction were found to be AFM. 
Doping carriers into such an AFM Kitaev-FM Heisen- 
berg model was also shown to stabilize the p SCi state 
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but such a state becomes unstable against a singlet SC 
state by further doping. Doping effects in the general KH 
model have not been studied, including the AFM Kitaev- 
FM Heisenberg interaction as alternatively suggested for 
Na2lr03. 

In this paper, we consider a general KH model in which 
both Kitaev interaction and Heisenberg interaction can 
be either FM or AFM. Doping effects are considered by 
introducing hopping terms which conserve isospin index 
a with the double occupancy prohibited as in the tJ 
model for high- Tc cuprates. The Hamiltonian is thus 
given by 

H^-tY,{4aCP>^ + H.c.)+HK + HH. (2) 

{rr') 

We investigate the global phase diagram of this model 
using an SU{2) SBMF method. We start from solving 
the undoped KH model defined on a finite cluster us- 
ing the Lanczos exact diagonalization method. We then 
introduce a mean-field decoupling scheme that can be ap- 
plied for both symmetric Heisenberg interaction and the 
anisotropic Kitaev interaction. Mean-field ansatze are 
constructed motivated by such exact solutions. Our re- 
sults demonstrate the clear difference between the AFM 
Kitaev model and the FM Kitaev model when carriers 
are doped, even though the undoped cases are equiva- 
lent. We confirmed novel triplet superconducting states 
reported previously. Yet, their relative stability is found 
to depend on the sign of the Kitaev interaction and 
the competition between the Kitaev interaction and the 
Heisenberg interaction. Additionally, s-wave and d-wave 
superconducting states in the AFM Heisenberg limit and 
the FM state in the FM Heisenberg limit are found. Our 
results could become guidelines for a materials search to 
realize specific properties and further theoretical analy- 
ses. As the present model is simple, testing or refining 
the current results by using more sophisticated methods 
is also possible and desirable. 

The rest of this paper is organized as follows: In SecHIl 
we examine the undoped KH model by using the Lanczos 
exact diagonalization method. The results are useful for 
selecting mean-field ansatze to be used later. A mean- 
field method is introduced in Sec. lIIIi and our results are 
presented in Sec. IIVI Section |V] is devoted to summary 
and discussion. 



II. UNDOPED CASE 

We first analyze the undoped KH model in detail. This 
analysis will be helpful for considering mean-field ansatze 
and understanding the phases arising by carrier doping. 

Before going into the detailed analysis, it is instruc- 
tive to perform the four-sublattice transformation.-^ii^ 
The four-sublattice transformation leads to the change 
in the sign of the Heisenberg term with Jh — t- —Jh 
and Jk —>■ Jk + 2 J//. When the Kitaev term and the 



Heisenberg term have the different sign, Jk vanishes 
at Jk = —2Jh- As the resulting Heisenberg model is 
AFM for the FM Heisenberg case and FM for the AFM 
Heisenberg case, the spin ordering around Jk — —'^Jh is 
"zigzag AFM" for the former and "stripy AFM" for the 
latter. Due to the larger quantum fluctuation, the total 
"staggered spin" in the rotated spin basis is reduced, and 
the parameter regime for this zigzag AFM is expected to 
be wider than that for the stripy AFM. When the Ki- 
taev term and the Heisenberg term have the same sign, 
the cancellation does not occur in Jk- Thus, a direct 
transition is expected between the Kitaev SL in the large 
\Jk\ regime and other competing phase stabilized in the 
large \Jh\ regime: the Neel AFM or the FM. 

We now employ the Lanczos exact diagonalization for 
the Hamiltonian for the undoped KH model Hk + Uh 
defined on a 24-site cluster with periodic boundary condi- 
tion. This cluster is compatible with the four-sublattice 
transformation^ which changes the original spin S to S. 
Numerical results for squared total spin and the nearest- 
neighbor (NN) spin correlations are shown in Fig. [51 As 
expected, there are two phases, Neel AFM and SL, for the 
AFM Kitaev- AFM Heisenberg case (upper left) and three 
phases, FM, zigzag AFM, and SL, for the AFM Kitaev- 
FM Heisenberg case (upper right). The phase boundaries 
are also signaled as peaks in the second derivatives of the 
total energy (not shown). In both cases, the SL regime 
is rather narrow with the nearly identical critical value 
\Jk,c\ ^ 0.98 separating it from magnetically ordered 
phases. For the AFM Kitaev-FM Heisenberg case, the 
phase boundary between zigzag AFM and FM is shifted 
from the classical value Jk = 1/2 to a smaller value 
Jk ^ 0.4 as discussed above. For the FM Kitaev case, 
the situation is just opposite to the AFM Kitaev case 
with the Neel ordering replaced by FM and the zigzag 
AFM by the stripy AFM. Here, the phase boundary be- 
tween the Neel AFM and the stripy AFM is shifted from 
the classical value Jk = —1/2 to Jk ~ —0.57. 

It is noted that the AFM Kitaev interaction is more 
destructive for the FM ordering than the FM Kitaev in- 
teraction for the FM ordering. All phase boundaries are 
consistent with the recent report in Ref. [H as obtained 
from the second derivative of the total energy. 



III. SLAVE-BOSON MEAN-FIELD THEORY 

In this section, we introduce a SBMF method that can 
be applied for both Heisenberg and Kitaev interactions. 
As usual, an S — 1/2 isospin operator is described by 
fermionic spinous as Sj = l/fg-Tjo-'/™-' with the local 
constraint '^^r fpaf^a- = 1, which is normally approxi- 
mated as the global constraint, t'^ is a Pauli matrix. 

In order to deal with the doping effect near a Mott 
insulating state excluding the double occupancy, two 
bosonic auxiliary particles 61^2 are introduced as cpa =^ 
Mbljra + fTfet^/t) (Ref. Mj with the SU{2) singlet 
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FIG. 2; (Color online) Lanczos exact diagonalization results for squared total spins (normalized to their values in the fully 
polarized FM state) and the NN spin correlations obtained on 24-site clusters as a function of Jk with Ju ~ ±(1 — |>/if|)- (a) 
AFM Kitaev case Jk > and (b) FM Kitaev Jk < 0. Solid (dashed) lines correspond to original (rotated) spin basis. Vertical 
dash-dotted lines are first-order phase boundaries. For each case, a magnetic pattern is schematically shown. 
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with 
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(3) 



(4) 



The global constraints {K'^) — are imposed by SU{2) 
gauge potentials a^. Doped carriers can be either holes 
or electrons. As the current model has only NN hop- 
pings [see Eq. there exists particle-hole symmetry 
about the zero doping, therefore the effect is symmetric. 
Focusing on the low-doping regime at zero temperature, 
we assume that all bosons are condensed, so that S — 
T,AblfrKp) « Ei^KMP and (KreA) = {±i){Kf»eB)- 



Imaginary number i appears when the Bose condensa- 
tion acquires the sublattice-dependent phaseJ^ 



A. Decoupling scheme 

In order to apply the SBMF method for both AFM and 
FM Kitaev interactions and AFM and FM Heisenberg in- 
teractions, we employ the decoupling scheme introduced 
in Ref. [isl Here, a spin quadratic term is decoupled into 
several different channels as 
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07 C7 A* A I * 4_ 4-1* 4-1 \ pi* pi ^ 



l_i V (t'l*tl'^ + el* el- ' 

I g / J \ rr' rr' rr' rr' . 



(5) 
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where A^p, = fraiT^aJpa' (singlet pairing), tl^, = 
JPa[iT'^'ry]„„,fp,„, (triplet pairing), Xrr' = fl^fr'a' (spin- 
conserving exchange term) , and e~, = fp^rj^jfa' (spin- 
nonconserving exchange term). Summation over 7 in 
Eq. ([5|) gives a Heisenberg term. Then, terms having the 
negative coefRcient are kept and the mean field decou- 
pling is introduced to them. This recovers the previous 
mean-field schemes.—"— Difi^erent decoupling schemes 
are also used in literature .^^i^^'^^ 

In what follows, we use the simplified notation in which 
the subscript fr' is replaced by the bond index p = x,y, z 
connecting the sites r G A and f' G B, for example, 
iXrr') for r' — r = Tp is written as (xp)- is a unit 
vector connecting the nearest-neighboring sites along the 
p bond as shown in Fig. [TJ These are explicitly given by 

= (-%/3/2, -1/2), ry = (^3/2, -1/2) and r, = (0, 1). 



B. Mean field Hamiltonian 



After the mean-field decoupling, the single-particle 
Hamiltonian is expressed as 



k 

(6) 

Here, a Nambu representation is used with 4-component 
spinors given by = (/J^^, /|^^, f_j:^^, /..-^J. 

Ht^K,H are 8x8 matrices. Ht includes both hopping 
terms and the chemical potential or the gauge field and 
is given by 



Ht{k) 



where 



(a^-ia^)e„'^ a'' 5„„> -X*{-k)S„, 



(7) 



(8) 



and £-|-^ ~ ^^It — 1 is the antisymmetric tensor. The prefactor ^S{i) for t comes from the mean-field decoupling for 

the bosonic term {bAib^si ^ ^^2^32)- 

Spin-spin interaction terms are both expressed as 



HK^k) = \Jk,h\ 



Xcra' 

Xl'jk) -A,'A-k) 

-KA-k) -x:>A-k) 



K'Ak) 



'XlA-k) 



(9) 



For the AFM Kitaev interaction, the matrix elements are For the AFM Heisenberg interaction, we have the well 



given by 



^(^) = g E^*'"''^^'')*^' - -^e*^~''(iP*f^f^(ll) 



known expressions 



(14) 
(15) 



with f° being the 2x2 unit matrix and, for the FM 
Kitaev interaction, these are given by 

*(^) = -^E(i -^P7)e*''''(er)^"' (12) 

^(^) = ~\ E(l - 5p,)A^-'^{t^p)iryT\ (13) 

P7 



while, for the FM Heisenberg, we have 



(16) 
(17) 



P7 
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Hq is a constant term for which the contributions 
from the AFM Kitaev and the FM Kitaev are given 
by lEp\JKmXp)\' + + |(A,)p + l(tpp) and 

lEp^lW - '5p^)(|(epP + l(tpP), respectively, and 
the contributions from the AFM Heisenberg and the FM 
Heisenberg are given by | Ep l-^ff I (KXp) P + K^p) H and 
lEp^ \JH\{\{eJ)\' + |(tpn, respectively. ^ 

Mean-field Hamiltonians shown in this subsection 
might become useful for refining the results to be pre- 
sented by using variational techniques. In principle, one 
can construct variational wave functions by (1) diago- 
nalizing mean-field single-particle Hamiltonians without 
contributions from slave bosons and (2) projecting out 
the unphysical doubly occupied states. Then, the total 
energy is computed by using thus constructed variational 
wave functions and is minimized with respect to varia- 
tional parameters. 



C. Mean-field ansatze 

Undoped Kitaev limit. The undoped FM Kitaev model 
was studied using the SBMF theory in Ref. [H, and the 
undoped AFM Kitaev model was studied in Ref. [l^. 
As expected from the true ground state of the Kitaev 
model which does not depend on the signs of exchange 
constants,— the two cases are shown to give the identical 
excitation spectrum. 

Using the current definition, the mean-field solution 
for the FM Kitaev model is given by — i(i^) = —i{ty,) — 
(t^) = {n) = {<) = (ep = 0.3812Z and {t%) = -i{ty) = 
(el) = -0.1188i. The mean-field solution for the AFM 
Kitaev model is given by —{xxyz) = = {t%} — 

i{ty) = 0.3812Z and -{el) = -(ep = (tp = (if) = 
i{ty) = i{ty) = -0.1188i 

As discussed in detail in Ref. ,22a the first mean-field 
ansatz describes a Z2 SL. The second ansatz uses the 
same gauge used in Refs. [l3 andlH, where the dispersive 
Majorana fermion mode is given by = "73 ^-^"^ '''' f\)- 
Thus, the mean-field ansatz for the AFM Kitaev model 
also describes a Z2 SL. 

In doped cases, a mean-field Hamiltonian has addi- 
tional three gauge potentials. With possible magnetic 
orderings, a total of ^ 30 parameters have to be deter- 
mined self-consistently. In order to make the problem 
tractable, we focus on the following five ansatze. The 
first four ansatze respect the sixfold rotational symmetry 
of the underlying lattice. 

p SCi. This mean-field ansatz is adiabatically con- 
nected to the mean-field solution for the Kitaev limit as 
described above. Here, the relative phase ±i is required 
between the Bose condensation at sublattices A and B 
with the SU{2) gauge potentials = — a^.^'* Because 
of this constraint, the spinon density {fl^ff^) differs from 
the "real" electron density (ct^Cf^) in the p SCi phase 
and a normal phase {{tj) = (ej) = (Ap) = 0) adjacent 
to it. In many cases, such a normal phase has slightly 



lower energy than the other SC ansatze, but this is an 
artifact of the constraint. In this work, we identify the 
upper bound for the p SCi phase as the smaller S where 
the order parameters for the p SCi phase become zero or 
the p SCi phase becomes higher in energy than the other 
phases. 

p SC2- The second ansatz is also a p SC. We assume 
the form of order parameters based on the leading pairing 
instability in the stability matrices M^^y^z (Refs. Tl5ll24l ) 
'AS d — dx + dy + dz. Here, d^ — {t2,t'^,t2), and we 

take dx = e*^=" (^i, ^2, ^3), dy = e'^" (ta, ii, ^2), and d^ = 
e^^' {t2,t3,ti) with ^1.2,3 being real. All solutions with 
— d.y, — or TT for 7 7^ 7' are found to degenerate and 
are lower in energy than the other combinations for both 
the AFM Kitaev and the FM Kitaev cases as reported 
in Ref. [l^ The details of the stability matrices and the 
symmetry of the order parameters are given in Appendix 

El 

s sc. The third ansatz is a singlet SC with the s wave 
paring. The SC order parameter is symmetric as (A^) = 
{Ay) = (A,) = A. 

d + id SC. The fourth ansatz is also a singlet SC with 
the d.j.i_y2 -\- idxy pairing (in short d + id pairing) .^-^ The 
spatial dependence of the SC order parameter is given by 
(A„A„A,) = A(e-2W3^e2-/3,l). 

For the latter three ansatze, we further introduce the 
following conditions: (1) Order parameters (e^ are as- 
sumed to be zero because these indeed become zero at 
large dopings and the fermionic dispersion relations gen- 
erally break the hexagonal symmetry when both (ej) 
and pairing order parameters {tJ) or (Ap) are finite. (2) 
The Bose condensation does not introduce a phase fac- 
tor. (3) The exchange term is symmetric (xp) = X and 
real. Thus, these ansatze are regarded as BCS-type weak 
couphng SCs. 

FM. Additionally, we consider the FM state. Here, 
we also introduce the local moment m = {J^J^ — f\f\) 
as a mean-field order parameter to represent the FM 
long-range order. When this order parameter is finite, 
site-diagonal terms in the mean-field Hamiltonian have 
\{?>Jh + JK)mT^^, for the AFM Kitaev-FM Heisenberg 
case and jJHmr^^, for the FM Kitaev-FM Heisenberg 
case, with Hq modified accordingly. The difference be- 
tween AFM Kitaev and FM Kitaev accounts for the fact 
that the FM Kitaev alone does not stabilize the FM long- 
range ordering but the AFM Kitaev coupling competes 
with the FM long-range ordering strongly. The choice of 
the spin axis can be taken arbitrary because of the spin 
rotational symmetry. But, with the current choice, the 
sixfold rotational symmetry is explicitly broken. 

Except for p SCi, the gauge potentials a^'^ = while 
7^ 0, thus the gauge symmetry is broken from SU{2) 
to Uil). 
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FIG. 3: (Color online) Mean-field phase diagrams for the doped Kitaev-Heisenberg model as a function of S and Jk with 
Jh = ±(2 - \ Jk\)- (a) AFM Kitaev {Jk > 0) and (b) FM Kitaev {Jk < 0). The phase boundary between the trivial p SC2 
and the topological p SC2 in (b) (light dash-dotted line) is obtained with reduced SC order parameters as discussed in the main 
text. We also plot phase boundaries at 5 — obtained from the exact diagonalization as open stars. 



IV. RESULTS 

A. Phase diagrams 

Schematic phase diagrams for the KH model are shown 
in Fig. 131 Here, to see various phases clearly, we chose 
the interaction strength as \Jk\ + \Jh\ — 2t. In what 
follows, t is taken as the unit of energy. 

For both the AFM Kitaev and the FM Kitaev cases, 
singlet SC states appear in the AFM Heisenberg side, 
d + id at small 6 and s at large S, and FM states in 
the FM Heisenberg side. The difference between the 
AFM Kitaev and the FM Kitaev is most visible near 
the Kitaev limit, where doping-induced p SCi states be- 
come unstable against the d + id SC for the AFM Kitaev 
rather quickly and against the p SC2 for the FM Kitaev. 
The d + id SC is continuously extended from the AFM 
Heisenberg limit, while the p SC2 for the FM Kitaev is 
only stable near the Kitaev limit. Further, the p SC2 for 
the FM Kitaev is more extended to the smaller doping 



regime than the d + id for the AFM Kitaev. This differ- 
ence can be understood from the different channels into 
which the Kitaev interaction is decoupled [Eq. ([5])] . For 
the AFM Kitaev, the singlet channel is weaker than the 
AFM Heisenberg by a factor of 3. On the other hand, for 
the FM Kitaev, the triplet channel is dominant as two 
components add up for one bond, for example and 
for 7 = z. Moreover, the doping-induced kinetic energy 
is better gained for the p SCi with the AFM Kitaev in- 
teraction because of the exchange term x which is absent 
in the FM Kitaev interaction. As the AFM Kitaev inter- 
action is decoupled into both the singlet and the triplet 
channels, the p SC2 could also be stabilized in the AFM 
Kitaev case. This happens when the singlet tendency is 
reduced by the finite FM Heisenberg interaction. 

It is noted that the phase boundary between the p SCi 
and the FM {d + id SC) for the AFM (FM) Kitaev case 
intersects the horizontal axis in the middle of the zigzag 
(stripy) AFM phase. This is expected because all states 
used to construct the phase diagram do not break the 
sublattice symmetry. When the zigzag and the stripy 
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AFM states are considered, these states should also be 
stabilized near the regimes indicated by the exact diag- 
onalization analyses. However, such states with longer 
periodicity are expected to be destabilized immediately 
by carrier doping as is the Neel AFM. Interestingly, the 
mean-field boundary between the p SCi phase and the 
d + id SC phase for the AFM Kitaev-AFM Heisenberg 
model in the limit of (5 — >■ agrees with the exact result 
on a finite cluster rather well [Fig.|3](a), left panel]. This 
may indicate that the uniform resonating valence bond 
(RVB) state at J = (singlet SC order parameters be- 
come exponentially small for both s SC and d + id SC 
states) is a good approximation for the Neel AFM state 
on a honeycomb lattice. 

In Ref. '22I, the quantum phase transition between the 
Kitaev SL and the FM for the undoped FM Kitaev-FM 
Heisenberg model (or equivalently between the Kitaev SL 
and the stripy AFM for the undoped FM Kitaev-AFM 
Heisenberg model) was studied using the SBMF approx- 
imation. There, the phase boundary between the Kitaev 
SL and the FM is shown to be located at Jk/Jh 4, 
which is consistent with the current result [see Fig. |3] (b) , 
right panel]. 

In the following subsections, detailed discussions on 
the p SCi and p SC2 phases and the relative stability 
between the s SC and d + id SC phases are presented. 



B. p SCi 

As discussed in Ref. [13 for the doped FM Kitaev 
model, the p SCi phase is characterized by the disper- 
sive x° Majorana mode and the weakly dispersive yf-y-'^ 
modes. 

Typical dispersion relations of the Majorana fermions 
are presented in Fig. 2] for various choices of parameters. 
In the undoped Kitaev limit (al), only the gapless x° 
mode is dispersive for both the AFM and FM. With finite 
Jh (bl,cl,dl), x''^'^ modes become dispersive while the 
X° mode remains gapless. 

At finite doping (S, ^^^'^'^ modes become dispersive and 
the x" mode is gapped. All modes are gapped by the 
mixing between different Majorana modes due to the fi- 
nite gauge potential a^^^ . For the FM Kitaev interaction 
with 5 = 0.02 (a2,b2,c2), the gap amplitude is ~ 2 x 10"*^ 
and is, therefore, invisible in Fig.|31 The finite gap in the 
X° mode results in the finite Chern number, -1-1 at the 
low doping limit. Softening of the ^^'^'^ modes is in- 
creased with the increase in 5. However, the softening is 
not strong enough to close a gap for the FM Kitaev inter- 
action before the p SCi phase becomes unstable against 
the p SC2 phase. Thus, the Chern number remains -1-1. 

For the AFM Kitaev interaction, we do see the strong 
softening of the yf^y^^ modes (d2). However, gap closing 
needed to change the Chern number from -1-1 takes place 
at relatively large Heisenberg interaction \ Jh/ Jk\ > 0.6 
and large doping 6 > 0.1. For such parameters, the cur- 
rent ansatz may not be a good approximation for the true 



ground state and/or the SU{2) SBMF method may not 
be reliable. 

For the FM Kitaev model, we notice that the soften- 
ing of the x^'^''^ modes in this work is weaker than that 
reported in Ref. [T^ . This is supposed to originate from 
the level of the mean-field decoupling. The current de- 
coupling is done in terms of spinous, while in Ref. [3 it is 
done in terms of Majorana fermions. Thus, it is possible 
that some order parameters, which are dropped off in the 
current scheme, are retained and have significant contri- 
butions. It is also noted that the x° mode and the x^'^''^ 
modes are shown to overlap at the M points in Refs. [l^ 
andll^ as in the current work, while they do not overlap 
at the M points in Ref. [13. Including these differences, 
further analyses might be necessary to fully understand 
the nature of the p SCi phase. 

Despite the subtlety in the mean-field scheme, the cur- 
rent study provides the "missing link" between the pre- 
vious results in Refs. 'l^ and ^3 near the FM Kitaev 
limit. The former describes the small-doping regime cor- 
rectly, while the latter describes the large-doping regime. 
Therefore, the first-order transition between the two is 
expected unless other phases intervene. In the current 
study, the first-order transition takes place at rather 
small dopings. The instability of the SCi phase comes 
from its inability to gain the kinetic energy by carrier 
doping because x is absent in the mean- field decouplings. 
As a result, the total energy has a positive slope as shown 
in Fig. [5|(a) . Similar phenomena appear to be happening 
in Refs. 14; in Fig. 4, the order parameter uo remains 
constant within the SCi phase. On the other hand, for 
the AFM Kitaev case, the p SCi phase benefits from the 
carrier doping like the d-wave SC in the tJ model, and 
the total energy shows a normal behavior [see Fig. [6] (a)]. 
In Figs. [5] (a) and [6] (a), one can see precursors of the 
unphysical behavior of the normal phase adjacent to the 
p SCi phase; i.e., the sudden decrease in the total en- 
ergy when the SC order parameters disappear. For the 
AFM Kitaev, this behavior starts to preempt transitions 
from the p SCi to the d + id or p SC2 by the finite FM 
Heisenberg interaction. A more reliable method such as 
variational Monte Carlo is necessary to locate the critical 
upper doping for the p SCi phase more accurately. 



C. p SC2 

Based on the analysis on the d vector^ there are 
three possible phases within the p SC2 regime: time- 
reversal symmetric (TRS) even-parity trivial phase, TRS 
odd-parity trivial phase, and TRS odd-parity topologi- 
cally nontrivial or topological phase. In our model, all 
these phases could appear depending on the interaction 
strength and the doping concentration. 

With the choice of 6*^ = 0, our triplet order parameters 
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0.30 




FIG. 5: Comparison between the p SCi and the p SC2 for the 
FM Kitaev with Jk = ~2 and Jh = 0. The total energy E 
(a) and order parameters (b) as a function of 5. t2 is defined in 
Eq. (|20|) . The hght vertical line in (a) indicates the boundary 
between the p SCi phase and the p SC2 phase. 



FIG. 6: Comparison between the p SCi and the d + id SC for 
the AFM Kitaev with Jk = 2 and Jh = 0. The total energy 
E (a) and order parameters (b) as a function of 5. The light 
vertical line in (a) indicates the boundary between the p SCi 
phase and the d + id SC phase. 



are expressed as 

{t:) = {ti)^{ti)^h, (18) 

(tt) = (tf.) = (tl) = (tl) = {tl) = {tD = i2 (19) 
for the AFM Kitaev-FM Heisenberg model and 

{tl) = "{t:) = ^{ti) - {tl) = {tl) = -{tl) - <2 (20) 

for the AFM Kitaev-FM Heisenberg model. 

For the AFM Kitaev case, the triplet SC order parame- 
ters are rather small as shown in Fig. [7] (a), and therefore 
the interband pairing can be neglected. At small dopings, 



there are four TR invariant k points (Mi. 2, 3 and F) be- 
low the Fermi level, thus this SC state is in the TRS 
odd-parity trivial phase. Phase transition takes place at 
5 ~ 0.25, above which only one TR invariant k point (F) 
exists below the Fermi level, to the topologically nontriv- 
ial SC in the class DII L^^i^^ This transition is signaled by 
the gap closing with the SC order parameters remaining 
finite as shown in Fig. [7] (b). For the AFM Kitaev case, 
the choice of phases O^^y^z = is found to correspond 
to the d vector rotating around the (1,-1,1) direction 
(see Appendix [B]). This corresponds to k^ — iky pairing 
for spins pointing in the (1, —1, 1) direction and k^ -\- iky 
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FIG. 7: 5'(7(2) SBMF results for the p SC2 phase in the doped 
AFM Kitaev-FM Heisenberg model with Jx = 1.3 and Jh = 
—0.7. (a) Order parameters and (b) SC gap amplitude as a 
function of doping concentration S. 



pairing for spins pointing in the (—1,1,-1) direction as 
in the B phase of superfluid '^He. 

For the FM Kitaev case, the situation was found to 
be more complicated because the interband pairing has 
finite contributions, as the triplet SC order parameters 
are much larger than those in the AFM Kitaev case as 
shown in Fig. [8] (a). When the SC order parameters 
are artificially reduced as (i^ ^(^p) with r < 1, a 
clear transition can be seen between the TRS odd-parity 
trivial phase at S < 0.25 and the TRS odd-parity topo- 
logical phase at S > 0.25 signaled by the gap closing 
[see Fig. [5] (b)]. As the order parameters are gradually 
increased, an additional transition shows up at small S, 
indicating the appearance of the TRS even-parity trivial 
phase. When the order parameters are fully developed, 
the TRS odd-parity trivial phase is overcome by the TRS 
even-parity phase, and the TRS even-parity phase di- 
rectly transitions to the odd-parity topological phase. 
Thus, as a function of temperature, the sequence of phase 
transition could appear within the mean-field approxima- 
tion, although only phase transitions at zero temperature 
are meaningful for two-dimensional systems. As for the 
AFM Kitaev case, the choice of phases 9x,y,z = cor- 
responds to the d vector rotating around the (-1,-1,1) 
direction in the TRS odd-parity phases. 

D. s SC versus d 4- id SC 

As discussed in Ref. [U, i J-type models on a honey- 
comb lattice have some preference for the d + id SC 



FIG. 8: SU{2) SBMF results for the p SC2 phase in the doped 
FM Kitaev model with Jk = ~'2 and Jh = 0. (a) Order 
parameters and (b) SC gap amplitude as a function of doping 
concentration 5. In (b), gap amplitudes obtained by using 
artificially reduced SC order parameters as (tj) =^ r{tj) with 
r < 1 are also shown with various r indicated. For r < 
0.4, there appear two gap minima, indicating the sequence 
of transitions from the even-parity trivial phase (small 5) to 
the odd-parity trivial phase (intermediate 5) and to the odd- 
parity topological phase (large S). 



over the s SC in the weak-coupling limit or near the 
critical temperature because of the interference between 
singlet pairing on different bonds. For the actual tJ 
model excluding the double occupancy, the stabilization 
of the d + id state was recently reported by using the 
Grassmann tensor product state approachi^ A similar 
effect was observed for an electronic model with repul- 
sive interactions."- 

Within a slave-boson mean-field approach, the relative 
stability between d+id and s SC states is rather subtleJ^ 
In Fig. [21 we compare the d+id SC and the s SC states for 
the doped AFM Kitaev- AFM Heisenberg model. As seen 
from the E-vs-5 curve, the d + id SC state is stabilized 
at smaller S regime, and the s SC state is stabilized at 
larger 6 regime. The s SC state has the larger SC order 
parameter A, while the d + id SC state has the larger x- 
This indicates that the kinetic energy is better gained in 
the d + id SC state, leading to its stabilization at small 
dopings. 

In Fig. [21 the d + id SC state is shown to be stabilized 
near the Kitaev limit compared with the s SC state. This 
is because the singlet pairing strength is reduced as one 
moves away from the AFM Heisenberg limit. The FM 
Kitaev interaction is more effective to reduce the paring 
strength. As a result, the d + id SC state is extended 
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FIG. 9: Comparison between the d + id SC and the s SC for 
the AFM Kitaev-AFM Heisenberg model with Jk = 0.8 and 
Jh = 1-2. The total energy E (a) and order parameters (b) 
as a function of 5. The light vertical line in (a) indicates the 
boundary between the d + id SC phase and the s SC phase. 



to larger dopings. This consideration also explains why 
the s SC state is extended to the lower doping regime 
in Ref. [Tsl . There, spin-conserving exchange terms x 
not considered for mean- field order parameters. Thus, 
the kinetic-energy gain by the Heisenberg term is under- 
estimated for the d + id state. 



V. SUMMARY AND DISCUSSION 

To summarize, we explored the possible novel phases 
induced by carrier doping into the KH model by using 
the SU{2) SBMF method. Various mean-field ansatze 
are motivated by the exact diagonalization results of the 
undoped model defined on a finite cluster. It is shown 
that the AFM Kitaev model and the FM Kitaev model 
are rather different when carriers are doped, although the 
ground state of the Kitaev model does not depend on the 
sign of the interaction, whether it is AFM or FM. In both 
cases, the d+id SC state is stabilized in the AFM Heisen- 
berg limit, the FM state in the FM Heisenberg limit, and, 
near the Kitaev limit, carrier doping first induces triplet 
superconductivity, p SCi. With the AFM Kitaev interac- 
tion, p SCi becomes unstable against a singlet SC states 
with the d + id symmetry, while with the FM Kitaev 
interaction it becomes unstable against another triplet 
SC state, p SC2. p SCi state breaks the TR symmetry 
and has the finite Chern number; in the current case the 
Chern number -1-1 is rather robust. This state is found 
to be more stable with the AFM Kitaev interaction than 
with the FM Kitaev interaction. Not only for the FM Ki- 



taev interaction, but also for the AFM Kitaev interaction 
the p SC2 state is stabilized when the Kitaev interaction 
and the Heisenberg interaction compete. The p SC2 state 
does not break the TR symmetry, but within this phase 
a sequence of topological phase transitions could take 
place. For the AFM Kitaev case, the intraband pairing 
is robust and the topological transition is between the 
TRS odd-parity trivial phase and the TRS odd-parity 
topological phase. On the other hand, for the FM Ki- 
taev case, the interband pairing contributes when the SC 
order parameters are developed, and, depending on the 
magnitude of the SC order parameters, the topological 
transition could be between the TRS even-parity trivial 
phase and the TRS odd-parity trivial phase, between the 
TRS odd-parity trivial phase and the TRS odd-parity 
topological phase, or between the TRS even-parity triv- 
ial phase and the TRS odd-parity topological phase. 

In this study, we used ansatze which do not break the 
sublattice symmetry or the underlying hexagonal sym- 
metry. "Zigzag" AFM and "stripy" AFM phases are, 
therefore, not considered, as such complicated magnetic 
orderings are expected to be destabilized immediately by 
carrier doping. But it remains to be explored whether 
novel SC states are realized by carrier doping or other 
states outside the ansatze are realized in the parameter 
regime where the Kitaev and the Heisenberg interactions 
compete. 

It is an interesting and important question whether 
or not the present model can be realized in real mate- 
rials. As discussed in Ref. [H, the AFM Kitaev-AFM 
Heisenberg model could be realized in artificial TMO het- 
erostructures, e.g., a bilayer of SrIrOs grown along the 
[111] crystallographic axis, when the local Coulomb in- 
teraction is large enough. In this case, the Heisenberg 
interaction is relatively large compared with the Kitaev 
interaction, and therefore the possible SC state induced 
by carrier doping is of the d + id. 

For (topological) quantum computations, triplet SC 
states, p SCi or p SC2 in the nontrivial phase, are desired. 
To realize the topological p SC2 state, one should include 
the FM Kitaev interaction as the dominant interaction or 
the AFM Kitaev interaction with finite FM Heisenberg 
interaction to suppress the tendency towards the singlet 
formation. A2lr03 with A=Li or Na was originally sug- 
gested as a candidate for realizing the FM Kitaev inter- 
action. But, later it was experimentally shown to have 
zigzag AFM ordering, indicating the importance of the 
longer-range interaction or the Kitaev interaction is ac- 
tually AFM with the finite FM Heisenberg interaction. If 
the latter situation is realized, carrier doping may induce 
triplet SCs. Yet, even in this case, the carrier hopping 
term does not conserve the isospin. Therefore, the sta- 
bility of the triplet SC states depends on the strength of 
the isospin-nonconserving hopping. 
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Appendix A: Stability matrix for the p SC2 phase 



the AFM Kitaev and = (0, 1/^2, -1/^2) for the FM 
Kitaev. The stability matrices My^z and the eigenvec- 
tors for My,z with the largest eigenvalue, say dy and d^, 
can be obtained from and {t^,ty,tz), respectively, by 
cyclically exchanging components. Any linear combina- 
tions of dx^y^z give the same critical temperature. But, 
the stable pairing amplitude at low temperatures must 
be determined by solving the non-linear gap equations. 
Appendix B: d vector analysis for the p SC2 phase 



The symmetry of the superconducting order parame- 
ters at the critical temperature can be deduced by 
analyzing the stability matrices!^'-'* which are derived 
from the linearized gap equations. For the triplet super- 
conductivity p SC2, the stability matrices consist of three 
independent matrices corresponding to (fp, (ip and (fp. 
For (tp, the stability matrix Mx is given by 



Here, we consider both intraband d vectors (dn) and 
interband d vectors {di2) (Refs. USEE) for our doped KH 
models by expanding the exponents in the anomalous 
terms [Eqs. (fTTj) and (IT7|) ] in the mean-field Hamilto- 
nian around fc = 0. For the AFM Kitaev-FM Heisenberg 
model, the intraband pairing is found to be dominant 
and the d vector is given by 



Mx 



[Jk-Jh)B ^JhC -JhC 
{Jk — Jh)C —JrB —JhC 
{Jk — Jh)C —JhC —JhB 



(Al) 



for the AFM Kitaev-FM Heisenberg model and 



Mx = 



-JhB -{Jk 
-JhC —{Jk 
-JhC —{Jk 



Jh)C -{Jk 
Jh)B -{Jk 
Jh)C -{Jk 



Jh)C 
Jh)C 
Jh)B 



(A2) 



for the FM Kitaev-FM Heisenberg model. Here, B = 
A.p=pi and C = Ap^pi , with the matrix A given by 



A 



pp' 



/ taDh{e+/2kBTc 
V 2^ 



tanh(e_/2fcBTc; 
2el 



V3 



1 



V3, 



1 



dii = iOy — —kx - -ky, — —kx + -ky,ky j,(Bl) 

where D ^{{Jk - JH)ti + Jnh} with ti(2) = (ip for 
7 = {¥^)P- For the FM Kitaev-FM Heisenberg case, using 
the same procedure for Eqs. (|13p and ([TT]). we obtain 



V3, 



V3, 



dii = iD[ -kx —ky,--kx ^ky,-kx ), (B2) 



di2 = + + ^fc: 



+ 2^^^y 4 ^v' ^xky ), (B3) 



X sin(fc ■ fp — 9) sin(fc • fp/ — 9) 

sm]i{fi/kBTc) cos{k ■ fp — 9) cos{k ■ fpi — 
^ 2^ cosh(£+/2fcBTe)cosh(e_/2fcsTc) 



(A3) 



Considering a symmetric state with (%) being indepen- 
dent of the bond specie, e± is given by e± — ±|£(fc)| — /i 



and 9 = arg[e(fc)] with £{k) = —teffj^r 



Here, 



Uff = lSt+^{JK&{JK) + SJHQ{JH)}{x*) with e being 
the Heaviside function. The leading pairing instability is 
determined by the eigenvector with the largest eigenvalue 
of Eq. (jAll) or (IA2p . As C < 0, such an eigenvector is 
expressed as dx = {t%,ty,tz) = (■\/l - 2r]'^,—if, —rf ) for 



where D = ^{Jk + Jh)^ with t2 

-{tl) = {tV) = {tl) = -{ti). 

When the intraband pairing is dominant, the p SC2 is 
in the TRS odd-parity phase. The choice of 9x.y.z = 
above describes the d vector rotating around the 
(1, -1, 1)[(-1, -1, 1)] direction for the AFM (FM) Kitaev 
case. This corresponds to kx — iky pairing for spins point- 
ing in the (1, — 1, 1)[(— 1, —1, 1)] direction and kx + iky 
pairing for spins pointing in the (—1, 1, — 1)[(1, 1, —1)] di- 
rection as in the B phase of superfluid '^He. For the FM 
Kitaev case, the contribution from the interband pair- 
ing becomes large when the SC order parameters are de- 
veloped, resulting in the TRS even-parity phase in the 
small-doping regime. 
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